There is often a need with electronic systems to monitor network traffic. As such, devices have been developed that analyze data packets within network packet streams that are communicated across networks. These network communications can occur through a number of different systems and can include both wired and wireless communication links. The analysis of network packets can occur both off-line and in real-time. For off-line analysis of network packets, the network packets are stored and then later analyzed without providing real-time data network analysis. For real-time analysis of network packets, the network packets within the packet stream must be analyzed fast enough to keep up with the real-time flow of network traffic. As such, real-time analysis is more difficult to achieve than off-line analysis of network data packets.
One problem associated with analysis of network packets is the large number of packets that are communicated and the speed at which they are communicated. For example, many network systems utilize communication links that operate at speeds of 1-10 Gbps (gigabits per second) and above. And many network communication systems have large numbers of active communication links at a time. This problem of analyzing network packets in high volume and high speed networks is made worse because many devices that analyze network packets do so using filters and other parameters that can cause duplicate packets to be present in the packet streams being analyzed. For example, monitoring redundant communication links, monitoring both ends of a communication link, and mis-configuration of copy ports (e.g., SPAN ports on Cisco network switches) can lead to duplicate packets in network packet streams to be analyzed. These duplicate packets increase the bandwidth and processing speed needed by the analyzing devices to process packet streams.
One prior solution to this problem of duplicate packets is simply to provide off-line removal of duplicate packets followed by off-line analysis of captured network packets. For this off-line solution, packets within a network packet stream can be captured and stored. The captured packet data file can then be processed to remove duplicate packets. For this removal of duplicate packets, for example, length and MD5 sum for packets can be compared to the previous packets (e.g., previous four packets), and matching packets can be removed as duplicates. It is noted that MD5 is a known, large cryptographic hash algorithm, which can be used to generate large hash values. Once duplicate packets are removed, the packet data file can then be analyzed for various events and/or occurrences, as desired.
While this prior solution provides some ability to eliminate duplicate packets for off-line processing, it is desirable to provide real-time solutions that remove duplicate packets from network packet streams.